Papers with Multi-Head Random Initialization
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-task Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves performance in multi-task learning by diversifying the head matrices through Multi-Head Dropout and Multi-head Random Initialization. |
| Approach: | They propose a low-rank adaptive approach to fine-tune large language models by approximating weight updates through low-ranked matrices. |
| Outcome: | The proposed approach improves performance in multi-task learning while reducing memory usage and training time. |